Scalable file management for a shared file system

ABSTRACT

Managing a shared file system comprising a directory and files stored on a multiple storage devices shared by plural processing nodes, is provided. A plurality of parallel directory traversal records are partitioned into a plurality of ranges to allow for the records in each range to be written independently in parallel by plural processing nodes during parallel directory traversal. Parallel operations are performed comprising parallel directory traversal of all directory paths and files in the shared file system.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application is a continuation patent application of U.S.patent application Ser. No. 12/540,176, filed on Aug. 12, 2009, thedisclosure of which is incorporated herein its entirety by reference.

BACKGROUND

1. Field of the Invention

The present invention relates generally to file management. Inparticular, the present invention relates to file management in sharedfile systems.

2. Background of the Invention

Advances in disk storage technology allow file systems to manage largevolumes of data, with increasing number of files stored in a single filesystem. Typically a file manager traverses the file system for selectingcandidate files that may be of interest to a user. As users requireoperations such as backup to complete within an assigned window, often 8to 24 hours, candidate file selection performance becomes a limitingfactor on the number of files that can be managed by the file managerwithin a single file system. Therefore, administrators typically “split”collections of files into large numbers of smaller file systems,resulting in higher costs for provisioning and management.

Conventional techniques for candidate file selection have the filemanager traversing the file system directory structure and performing a“stat” operation on every file and every directory. The “stat” operationinvolves retrieving the attributes of a file or directory necessary todetermine if a file is a candidate. Typically, this is performed byreading the file “inode” which contains information about the file, suchas size and the last time it was changed. The inode does not contain thefile name. The file names are stored in a directory structure whichincludes file names and corresponding inode numbers. The stat is alsonecessary for locating sub-directories, which must also be traversed bythe file manager to locate all files in the file system.

Performing candidate selection in such a manner requires a minimum ofone stat operation for every file or directory. Typically, thedirectories impose an arbitrary order on the files, causing the statoperations to generate random small reads of file metadata. The filemanager has no knowledge of the number files and/or subdirectories thatreside beneath an arbitrary directory in the namespace.

BRIEF SUMMARY

A method is provided for managing a shared file system including adirectory and files stored on multiple storage devices shared by pluralprocessing nodes. In one embodiment the method comprises performingparallel directory traversal of all directory paths and files in theshared file system utilizing a node-wise parallel directory traversal.Each node maintains a local workload queue of elements, the elementsrepresenting a plurality of un-read directories. A master node monitorstraversal processing workload of the plurality of the nodes anddynamically re-balances the workload across the plurality of nodes basedon the monitored workloads.

Another embodiment for managing a shared file system comprisespartitioning a plurality of parallel directory traversal records into aplurality of ranges to allow for the records in each range to be writtenindependently in parallel by plural processing nodes during paralleldirectory traversal. A plurality of file inodes are scanned in parallelby reading, with pre-fetching, full storage units of inodes from each ofthe plurality of ranges of directory traversal records. A file inodecomprises file attributes, and the plurality of inodes represent all thefiles in the shared file system in inode number order in an inode file,such that the inodes read are specified by said records. Each set ofrecords comprising a range is sorted by inode number.

Further, a computer program product for managing a shared file systemcomprising a directory and files stored on multiple storage devicesshared by plural processing nodes, is provided. The computer programproduct comprises a computer usable medium having computer readableprogram code embodied therewith, wherein the computer readable programwhen executed on the computer causes the computer to perform paralleldirectory traversal of all directory paths and files in the shared filesystem utilizing a node-wise parallel directory traversal. A localworkload queue of elements is maintained at each node, the elementsrepresenting a plurality of un-read directories. A master node monitorstraversal processing workload of the plurality of the nodes, anddynamically re-balances the workload across the plurality of nodes basedon the monitored workloads.

Another embodiment comprises a data processing system, comprisingmultiple information storage nodes and multiple processing nodes, eachprocessing node comprising a central processing unit for executingcomputer usable program code. The system further comprises computerusable program code configured to perform a parallel directory traversalof all directory paths and files in a shared file system utilizing anode-wise parallel directory traversal, and maintain a local workloadqueue of elements at each node, the elements representing a plurality ofun-read directories, wherein traversal processing workload of theplurality of the nodes is monitored by a master node and dynamicallyre-balanced across a plurality of nodes based on the monitoredworkloads.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1A shows the architecture of an example computing environmentincluding computing nodes configured according to an embodiment of afile management system disclosed herein;

FIG. 1B shows the architecture of the computing nodes configuredaccording to a file management system disclosed herein;

FIG. 2 shows an example general parallel file system (GPFS) implementingthe disclosed file management system;

FIG. 3 shows an example inode space partitioned according to thedisclosed file management system;

FIG. 4 shows a flowchart of a process implementing the disclosed filemanagement system;

FIG. 5 shows a more detailed flowchart of a process implementing thedisclosed file management system; and

FIG. 6 shows a block diagram of an example computing system in which anembodiment of a file management system according to the invention may beimplemented.

DETAILED DESCRIPTION

The following description is made for the purpose of illustrating thegeneral principles of the invention and is not meant to limit theinventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc. The descriptionmay disclose several preferred embodiments for file management in ashared file system. While the following description will be described interms of such languages for clarity and placing the invention incontext, it should be kept in mind that the teachings herein may havebroad application to all types of systems, devices and applications.

Scalable file management for a shared file system is provided. Oneembodiment comprises a file management system for efficient sortingusing a file system itself to share data between processes running inparallel across several computing nodes in a storage cluster (e.g., FIG.1A, described further below). The file management system is scalable,wherein as the number of nodes increases the degree of parallelismincreases. This reduces the time required for candidate file selection.This scalable system enables file management for extremely large filesystems with a multitude (e.g., many millions and billions) of files.

In one example of the scalable file management, a large shared filesystem allows sharing data between processes running in parallel acrossa plurality of storage devices in a cluster, providing scalable filemanagement. File management comprises using a plurality of processingnodes for performing a parallel directory traversal of all paths andfiles in the shared file system stored on a plurality of storage devicesby a master node. Each processing node comprises a local work queue ofelements, the elements representing a plurality of un-read directories.A plurality of execution threads are implemented to read the elementsfrom the local work queue.

A plurality of parallel directory traversal records are partitioned intoplural buckets, wherein the records in the buckets are writtenindependently in parallel by the plurality of processing nodes duringparallel directory traversal. A plurality of file inodes is scanned inparallel by reading (with pre-fetching) full disk blocks or tracks ofinodes from a plurality of pre-assigned buckets. Each file inodecomprises file attributes, wherein the plurality of inodes representsall the files of the shared file system in inode number order in aninode file.

Metadata associated with the inode file representing all the files ofthe shared file system, is retrieved in inode number order andevaluated. If a processing node fails to perform an inode scan of apre-assigned bucket, the inode scan of the pre-assigned bucket isre-assigned to another processing node. The plurality of executionthreads may use a plurality of local mutual exclusions (mutexes) toresolve contention for access to a plurality of local process resources.The master node monitors a workload queue length of the plurality ofprocessing nodes and dynamically re-balances the workload across theplurality of computing nodes.

The parallel directory traversal of the shared file system is executedusing a node-wise parallel directory traversal, and may be executedusing a thread-wise parallel directory traversal. Further, the pluralityof parallel directory traversal records in the buckets may bepartitioned based on a sequence ordered by inode number.

An implementation of the disclosed scalable file management system usinga shared file system is described herein below. A general parallel filesystem (GPFS) conforming to the Portable Operating System Interface forUnix (POSIX) standard for file systems is utilized as the shared filesystem.

One embodiment of a computing environment incorporating and/or usingaspects of the present invention is described with reference to FIG. 1A.Computing environment 10 includes one or more nodes 12 (e.g., Node 1, .. . , Node N), which share access to one or more storage devices 14(e.g., Disk 1, . . . , Disk m). The nodes are coupled to each other andto the storage devices via an interconnect 16. In one example, theinterconnect includes a wire connection, token ring or networkconnection. One communication protocol used by one or more of theseconnections is TCP/IP. As one example, a node 12 includes an operatingsystem 20 (FIG. 1B), such as the AIX operating system, offered byInternational Business Machines Corporation (IBM). The operating systemincludes a file system 22 (e.g., a software layer), such as GPFS,offered by IBM, which is used to manage one or more files located in thevarious storage devices. In one example, each file is associated with adirectory and in particular, with a directory block of a directory. Adirectory includes one or more directory blocks, and each directoryblock has associated therewith zero or more files (a computingenvironment may include one or more directories). Further, each file mayhave associated therewith metadata that includes, for example, fileattributes, such as file size, last-modified time, and file owner. Thismetadata is contained within a data structure, referred to as an inode.

FIG. 2 shows a block diagram of an example of a file system managed byan embodiment of a scalable file management system disclosed herein. Thefile system includes directory tree 100, inode file 200 and data 300.These three elements are typically present in a file system as filesthemselves. For example as shown in FIG. 2, inode file 200 comprises acollection of individual records or entries 220. There is only one inodefile per file system. Entries in directory tree 100 include a pointer112 comprising an integer quantity which operates as a simple index intoinode file 200. Entries 217 are employed to denote a file as being adirectory.

A directory comprises a file in which the names of the stored files aremaintained in an arbitrarily deep directory tree. The directory tree isa collection of directories which includes all of the directories in thefile system. A directory is a specific type of file, which is an elementin the directory tree. A directory is a collection of pointers to inodeswhich are either files or directories which occupy a lower position inthe directory tree. A directory entry is a single record in a directorythat points to a file or directory. FIG. 2 shows an example directorytree within block 100, wherein a directory entry 120 includes file name111, inode number 112 and may further include a file/directory indicator113.

While FIG. 2 illustrates a hierarchy with only two levels, it should beunderstood that the depth of the hierarchical tree structure of adirectory is not limited to two levels (there may be dozens of levelspresent in any directory tree). The depth of the directory tree does,nevertheless, contribute to the necessity of multiple directoryreferences when only one file is needed to be identified or accessed.However, in all cases the “leaves” of the directory tree are employed toassociate a file name 111 with entry 220 in inode file 200. Thereference is by “inode number” 112 which provides a pointer into inodefile 200. There is one inode array in file systems of the typeconsidered herein. The inode array may comprise the inode file 200 andthe index points to the array element (e.g., inode #10876 is the10876^(th) array element in inode file 200).

Typically pointer 112 is a simple index into inode file 200 which isthus accessed in an essentially linear manner. Thus, if the index is10876, it points to the 10876^(th) record or array element of inode file200. Name entry 111 allows one to move one level deeper in the tree. Intypical file systems, name entry 111 points to, e.g. inode #10876, whichis a directory or a data file. If it is a directory, one recursivelysearches in that directory file for the next level of the name. Forexample, assume that entry 111 is “a,” as illustrated in FIG. 2. Onewould then search the data of inode #10876 for the name entry with theinode for “a2.” If name entry 111 points to data, one has reached theend of the name search. The name entry 111 may include an additionalfield 113 which indicates whether this is a directory or not. Thedirectory tree structure is included separately because POSIX allowsmultiple names for the same file in ways that are not relevant to eitherthe understanding or operation of the present invention.

Directory tree 100 provides a hierarchical name space for the filesystem in that it enables reference to individual file entries by filename, as opposed to reference by inode number. Each entry in a directorypoints to an inode. That inode may be a directory or a file. Inode 220is determined by the entry in field 112 which preferably is an indicatorof position in inode file 200. Inode file entry 220 in inode file 200 istypically, and preferably, implemented as a linear list. Each entry inthe list preferably includes a plurality of fields: inode number 212,generation number 213, individual file attributes 214, data pointer 215,date of last modification 216 and indicator field 217 to indicatewhether or not the file is a directory. Other fields not of interest orrelevance to the present invention are also typically present in inodeentry 220. The inode number is unique in the file system. The filesystem preferably also includes generation number 213 which is typicallyused to distinguish a file from a file which no longer exists but whichhad the same inode number when it did exist.

Inode field 214 identifies certain attributes associated with a file.These attributes include, but are not limited to: date of lastmodification; date of creation; file size; file type; parametersindicating read or write access; various access permissions and accesslevels; compressed status; encrypted status; hidden status; and statuswithin a network. Inode entry 220 also includes entry 217 indicatingwhether the file it points to is in fact a directory. This allows thefile system itself to treat this file differently in accordance with thefact that it contains what is best described as the name space for thefile system itself. Most importantly, however, typical inode entry 220contains data pointer 215 which includes sufficient information toidentify a physical location for actual data 310 residing in dataportion 300 of the file system.

The file management system provides: (1) the capability of rapidlyreading the inode file by block instead of by individual inode, allowingmore efficient use of disk storage and also allows overlapping of thereading of inode entries across multiple disks; (2) the capability ofsplitting the workload into pieces which results in approximately equalstresses on the backup target, enabling the workload to be splitaccording to the size of the file and also to be split in accordancewith the number of files and further allows the scheduling of “worker”threads to do the actual data movement to the backup target or targets;and (3) the capability of backing up data from the same portion of thefile name space to the same backup target every time, or for it tomigrate to other backup targets to better balance the backup workload.

It is assumed herein that there is a backup facility which accepts andstores backup copies of data. Tape management and cataloging features ofsuch a facility are used in the operation of the present invention;however, numerous examples of such products exist in marketplace. IBM'sTivoli Storage Manager is an example of one such product. IBM is alsothe assignee of the present invention. Relevant portions of thestructure of IBM General Parallel File System (GPFS), which is similarin many ways to any file system conforming to the XOPEN standards forfile systems, are now briefly considered in order to provide a contextfor a proper understanding of the present invention.

The starting point for the file system (i.e., shared file system) is afile system descriptor data structure, called the superblock, whichresides at a known fixed point on the disks which comprise the filesystem. This structure is required in order to perform any action withrespect to the file system. This structure has pointers to the disklocation of two primary data structures utilized by the scalable filemanagement system. The first of these primary data structure is inodefile 200. The inode file is a collection of individual inodes whichconstitutes the data structure that describes the key properties of thefile. This information is returned via the stat call. The entries in theinode file contain time stamps which reflect the last time that the fileor its properties have been changed and it also indicates the size ofthe file. The inode for a file does not contain the name of the file.The file names are stored in the file system directory structure 100.

The second primary data structure comprises the root directory 100 forthe file system, addressed by the file system superblock. The rootdirectory is structurally the same as any other directory in that itcontains a series of records including a name for a file or anotherdirectory, an inode number that points at an inode containing theproperties of the file and in GPFS, a target-type field that describeswhether the target of the directory entry is a file or anotherdirectory. A file which is named “rootdir/username/fileA” is located bysearching the directory called “rootdir” which points to the inoderepresenting the directory named “username”. The directory named“username” contains the inode number of the file named “fileA”. Innormal access operations, each directory in the path and itscorresponding inode must be accessed to find the data. Exampleoperations include reading directory entries and then (usuallyinterwoven) the inodes to which the directory entries refer.

The performance characteristics of the above-mentioned operations aremonitored. A directory is a file in most UNIX style systems, wherein theread operation on a directory involves reading the directory inode andreading the appropriate directory blocks. Each read is a disk operationunless the access pattern and/or the amount of caching available allowthe required data to be cached. Of specific interest herein are thosecases where the size of the file system and the requirement that themetadata be available to multiple systems, make it unlikely that therequired data be already cached.

There are two standard techniques used for backup. One of thesetechniques is “backup by inode.” In this technique, one reads modes inbulk from the inode file and backs up the data associated with eachinode file entry. The files are identified by inode number. This meansthat any attempt to restore an individual file by name involvessignificant custom programming that is not commonly done. Thealternative and more common technique is to scan the file system namespace looking for files which meet the backup criteria (e.g., modifiedafter a specified date). By using the name space, the program does aninquiry on each file in the entire name space. The standard interfacesto do this require a read of the directory entry and the inode for thefile; this process requires separate disk operations. This pass throughthe entire name space is extremely time consuming for larger filesystems.

According to the scalable file management system disclosed herein,candidate selection starts by directory traversal at the root directory,without requiring stats to determine the directory structure. Access tothe inode file is a parallelized sequential scan of the inode file.

The list of files generated from the directory traversal need not be ina completely sorted inode order for efficient parallel access to theinode file. Instead, the inodes can be partitioned into smaller buckets(ranges) of inodes. Utilizing said partitioning as the unit of work fora parallel scan, requires only the inodes within each range to besorted. The sorting within each range may be performed by a conventionalsort program. The number of inodes in each range can be adjusted so thatall the records to be sorted “fit” within a fast random access memory ofa computer, wherein sorting can be executed very quickly and withminimal input/output operation.

High speed operations for a file system, such as disclosed in U.S. Pat.No. 7,092,976, are performed in parallel on each range of inodes asdisclosed herein. For example, as described in U.S. Pat. No. 7,092,976,typed directory entries may be used, allowing the complete file systemnamespace to be traversed without performing a single stat operation.The namespace traversal allows the file management program to constructthe full path to every file by reading the only directory entries andnot the inodes. The path names do not contain the stat information abouteach file, but they do contain inode number of each file.

According to the invention disclosed herein, stats may be performedusing multiple (issued in parallel) sequential accesses to the filemetadata. This essentially allows the stat of every file to be performedusing sequential reads of large blocks of inodes, rather than randomreads of individual inodes. This is termed “parallel sequentialtraversal” herein. It is not required to sort the list of file namesinto the order they are stored on the disk (i.e., parallel sequentialtraversal does not require a total inode sorted order). Traversal isdivided into an arbitrary number of segments (buckets), wherein only theinodes within a segment must be in sorted order. The number of segmentsmay be selected to match a desired degree of parallelism and/or tominimize time to sort each segment.

In one implementation, parallel directory traversal is implemented bymultiple computing nodes and multiple threads at each computing node,with dynamic load balancing and redistribution of node queue entries.The results of the parallel directory traversal are distributed intoresult sub-buckets, indexed by a hash of the high bits of the inodenumber and the node number of the node that found the inode, whereinruns of consecutive inodes are kept together. The size of the bucketscan be kept modest so that subsequent steps can be performed withinavailable memory (e.g., dynamic random access memory) of a node,essentially eliminating external (I/O bound) sorting. This improvesparallel sorting since sorting all of the inode space is not required bythe system. Partitioning by hashing the high bits allows sorting eachbucket, requiring only a fraction of the entire inode space. Allsub-buckets for each hash result value are merged and sorted together,each one produced by a different node, providing all of the inodes inuse within the file system that are in a range of inodes. The inodeswithin each range are read, exploiting the efficiency and speed withwhich (nearly) consecutively numbered inodes can be read, because theyare arranged sequentially on disk tracks. Upon reading an inode, thepolicy rules or file selection criteria are evaluated. Said merging andsorting can be performed in parallel with, and pipelined to, the inoderead and policy evaluation.

Referring to FIG. 3, in one example, an inode space 300 is partitionedinto clumps or buckets (ranges) 302, so that there are long runs ofconsecutively numbered inodes (i.e., inode numbers) within each bucket.Each bucket (range) 302 is “spread” across N nodes 12, into Nsub-buckets (sub-ranges) 302S. Multiple (M) buckets 302 are used toallow operation thereon in parallel. Each inode number i (0<=i<[a verylarge integer]) is mapped to a bucket number j (0<=j<M) with a functionƒ having desired properties. Preferably, runs of inode numbers all fallinto the same disk blocks and disk blocks are allocated by powers of 2.As such, a number L is selected so that storage for 2^(L) consecutiveinodes span several disk blocks (sufficient to gain efficiency frompre-fetching and buffering built into inode scan mechanisms such as GPFSor similar file systems).

An example of said function ƒ can be of form:

-   -   ƒ:=g(i modulo(2^(L))) or    -   ƒ:=g(i>>L), which considering the binary representation of i,        shifts to the right and discards the L low order bits of i.

The function g can be any (hash) function that takes large integers andmaps them to said M bucket numbers ranging 0, . . . , M−1. In general gis selected to accommodate lack of apriori knowledge of the distributionof inode numbers actually used to represent files. For example, a simplefunction g(k)=k modulo M may be used.

Referring to FIG. 4, in conjunction with FIGS. 1-3, the scalable filemanagement system implements a process 400 including the followingprocess blocks:

-   -   Block 401: N computing nodes working in parallel traverse (walk)        the file system directories.    -   Block 402: Each directory entry is assigned to one of N×M        sub-buckets 302S based on the computing node number and the        inode number.    -   Block 403: The N nodes store entries into N columns×M rows of        sub-buckets 302S (typically M>N), with each node storing into        its own column of M sub-buckets 302S.    -   Block 404: In the next phase, nodes are assigned rows of N        sub-buckets 302S to traverse. Each row of N sub-buckets 302S        includes 1/M of all inodes.    -   Block 405: Each row of N sub-buckets 302S is sorted in inode        order for policy evaluation.    -   Block 406: Each node evaluates policy rules on the inodes in its        assigned rows.

As such, in a directory traversal phase, while executing a node-wise andthread-wise parallel directory traversal, each of N nodes canindependently store directory traversal records (DTRs) [inode number,pathname] into its own set of M sub-buckets. Thus, no node-to-nodecoordination or locking is required to store a DTR into a sub-bucketduring the directory traversal phase. At the end of the directorytraversal, there are N×M sub-buckets representing M rows of N columns,each column written by a different processing node and each rowrepresenting a complete bucket. An example of a DTR is a tuple (inodenumber, pathname).

Further, according to the scalable file management system, directorytraversal is distributed over N (several or many) computing nodes,wherein dynamic load balancing results in a factor N speedup overconventional file management methods.

In a scan phase comprising inode scan (and policy evaluation), each nodeaccesses a row of N sub-buckets which logically comprises a completebucket (as defined by the mapping function, f, above), and sorts all theDTRs in those sub-buckets into a sequence ordered by inode number. Assuch, the inode scan can most efficiently take advantage of theaforementioned pre-fetching and buffering. In addition, the -sub-bucketsmay be stored in a shared file system (such as GPFS) where each node canefficiently access any file.

Further, the number of buckets, M, can be selected to be large enough todivide the space of all inodes, such that the storage for the DTRscomprising any bucket can be sorted very quickly within the fast memoryavailable to a node. This avoids the expense of a large input/output(I/O) intensive sort.

Further, the results of said sort can be “piped” (in the classical Unixsense) to the inode scan process, thus avoiding further I/Os which wouldotherwise be required to write out and read back the sorted bucket ofDTRs.

Further, having partitioned the work and data into buckets, each bucketconstitutes a unit of work which is recoverable/restartable/redoable,whereby the inode scan (with policy evaluation) phase may befault-tolerant. More specifically, if the process assigned to scanningany particular bucket j fails in any way, the work of scanning bucket jcan be re-initiated on any available computing node.

The scalable file management system can be programmed to tolerate faultsin individual computing nodes, such that if some nodes fail during theprocess, work can be re-assigned to the “surviving” nodes. Afault-tolerant parallel directory scan is provided, and beyond GPFSpolicy evaluation, the scalable file management system can be used togreatly accelerate the execution of the classical UNIX/POSIX ‘find . . .|xargs . . . ‘ paradigm.

Parallel Directory Traversal

The scalable file management system quickens directory traversal bydistributing directory traversal to multiple threads of execution onmultiple computing nodes. The system eliminates conventional seriallyexecuted directory traversal that issues one stream of read commands toa collection of disks. The parallelized directory traversal disclosedherein issues multiple streams of essentially random read commands, thusexploiting available parallel (or concurrent) operation of many storagedevices (disks) over which a large file systems directory may be spread.Referring to FIG. 5, an example parallelized directory traversal isorganized according to a file system management (FSM) process 500. Eachcomputing node implements the FSM process 500 as FSM logic (FIG. 1A). Asshown in FIG. 5, the FSM process 500 includes process blocks 501-510,described herein below.

-   -   Block 501: A master node (e.g., Node 3, FIG. 1A) initiates and        controls parallel directory traversal.    -   Block 502: Each node maintains a local work queue comprising one        or more elements, each element representing a directory that has        not been read yet.    -   Block 503: At each node, multiple execution threads read from        the local work queue. The threads all operate within a single        process on a single node, and use local mutexes to resolve        contention for access to local process resources such as the        local work queue of directories and the output files        (sub-buckets of DTRs, described below). Each thread repeatedly        dequeues an element, reads all the entries of the directory        specified by said element, writes records (DTRs) describing each        directory entry into sub-bucket files, and enqueues any        sub-directory entries it reads onto the local queue of work        elements. The threads continue until all the paths and files        within the file system have been traversed.    -   Block 504: Each node reports its work queue length to the master        node when crossing a high or low threshold, and/or periodically.    -   Block 505: The master node monitors the queue lengths at each        node.    -   Block 506: The master node dynamically re-balances the workload        across all the nodes based on the monitored/reported work queue        length, by:        -   randomly shuffling its list of nodes;        -   scanning through the shuffled list of nodes looking for            nodes that are under-utilized or over-utilized (having            queues that are shorter than the low threshold or longer            than the high threshold);        -   sending work elements from the master node local queue (if            any are available) to the nodes that are under-utilized;        -   if there remain both under-utilized and over-utilized nodes,            the master node requests the over-utilized nodes to send            directory entries to the master node. When directory entries            are received by the master node, it places them in the            master's node local queue and restarts the re-balancing            process (the re-balancing process may also be restarted            whenever the master node receives a report of an            under-utilized node).

GPFS stores the inodes that represent all the files of a file system ininode number order in a special file. A GPFS inodes scan can achievehigh performance by reading (with pre-fetching) full disk blocks ortracks of inodes when scanning consecutive (or nearly consecutive) inodenumbers. The parallel directory traversal stores records (DTRs)containing [inode-number, pathname] for the files it discovers in amultitude of output files (sub-buckets of DTRs) that will be written andthen processed in parallel.

In order to fully utilize all computing resources (e.g., CPU, memory,I/O channels) while minimizing contention and minimizing use ofcluster-wide locks within the GPFS cluster during both the directorytraversal and the parallel inodes scan with metadata evaluation, the setof DTRs are partitioned into multiple sub-buckets. The DTRs within manyor several sub-buckets can be written independently and in parallel bythe GPFS computing nodes during the directory traversal.

During the inodes scan, the DTRs within many or several sub-buckets areindependently, and in parallel, read, merged and sorted by inodenumbers, with metadata evaluation. The sub-bucket files of DTRs arecreated within a directory of a file system that is set up for sharedread/write access among all the nodes of the GPFS cluster. Thisdirectory can be within the GPFS file system being scanned or withinanother shared file system. The sub-bucket files are given names of theform: PIL.bucket_number.node_number.

The set of DTRs from the several sub-buckets with an identicalbucket_number logically comprise one bucket of DTRs. Each DTR isassigned its bucket_number, wherein when all the records within all thesub-bucket files with the same bucket_number are merged and sorted byinode numbers into a single list. As noted, this sorted list of DTRs(records) will have long runs of nearly consecutive inode numbers. Asub-bucket 302S PIL.j.i is the subset of DTRs computed by node i, wherethe inode number of each DTR hashes to the number j. The j^(th) bucket302 is the subset of DTRs (computed by all nodes) where the inode numberof each DTR hashes to the number j. As such, bucket j=Union {PIL.j.i::for i ranging from 0 to N−1, wherein N is the number of nodes 12}.

Bucket numbers are assigned so that each bucket 302 has approximatelythe same number of records. A simple formula that in practice achievesboth objectives is:

-   -   Bucket_number=(Inode_number>>L bits) modulo Number_of_buckets    -   L bits is selected so that 2^(L) encompasses several I/O blocks        of inodes, sufficient to take advantage of I/O pre-fetching. For        an example GPFS, a typical number for Lbits is 16. The notation        i>>L means to consider the number i as a binary string and shift        out the rightmost L bits. Equivalently i>>L is i divided by        2^(L) with the fractional part (or remainder) discarded. These        are common notations in the C and Java (and other popular)        programming languages.    -   Number_of_buckets is selected to be large enough so that both: a        full bucket 302 of DTRs can be sorted in memory without the need        for temporary files or encountering excessive virtual memory        paging, and that there will be sufficient buckets to fully        utilize all CPU and I/O channels during a parallel inode scan.

In an example GPFS, to scan a file system with one billion files, M=511buckets are used, leaving about 2 million DTRs in each bucket. Duringthe parallel directory traversal, each GPFS node i, writes DTRs to itsown set of M output files (where M=Number_of_buckets). The sub-bucketfiles 302S are PIL.0.i, PIL.1.i, . . . , PIL.M−1.i. The following holdstrue for each file directory entry traversed during the paralleldirectory traversal process: A DTR representing a file directory entrytraversed by node i whose bucket number is j is written by node i tofile PIL.j.i. The directory traversal yields pathnames of the allocatedinode numbers of the file system.

Parallel Modes Scan with Evaluation of File System Metadata

Having found the pathnames of the allocated inode numbers of the filesystem (representing all of the files), one or more of the metadataassociated with said files are to be retrieved. Then, one or morefunctions are to be evaluated on their pathnames and metadata.Typically, such functions are “policy” or “storage management”functions, for example of the form: If a file pathname matches a patternand the metadata (or attributes) of the file (stored in the inode)indicate a condition, then perform an action.

The sub-bucket files of DTRs, written by the parallel directorytraversal as described above, become the primary input to a highlyparallel inodes scan process with evaluation of file system metadata.Said management process 500 in FIG. 5 further includes the followingprocess blocks for a parallelized inodes scan process:

-   -   Block 507: Each node spawns multiple parallel inodes scan        process threads.    -   Block 508: Each inode scan is performed on one bucket 302        (FIG. 3) of DTRs, assigned by the master node. Before beginning        the inodes scan of a bucket, each inodes scan thread first        reads, merges and sorts by inode number all the DTRs that        comprise its assigned bucket.    -   Block 509: Each inodes scan thread performs successive GPFS fast        inodes scans (with evaluation of file system metadata). Each        inodes scan is performed on one bucket 302 (FIG. 3) of DTRs,        assigned by the master node. As such, each scan thread        successively requests a bucket number from the master node, and        the master node replies by doling out a bucket number j that has        not yet been assigned. The scan thread performs an inode scan on        the inodes represented by the DTRs within the assigned bucket        number j. This process continues until all buckets have been so        processed.    -   Block 510: Upon node failure, master node re-assigns        corresponding buckets to other inode scan threads on other        nodes.

To best exploit the GPFS inodes scan facility, just before beginning theinodes scan of a bucket j, an inodes scan thread first reads, merges andsorts by inode number all the DTRs that comprise its assigned bucket j.These are the sub-bucket files named: PIL.j.0, PIL.j.1, . . . ,PIL.j.N−1, where N is the number of computing nodes that executed theparallel directory traversal process. On a POSIX or UNIX-like systemthis can be accomplished by spawning a process to execute a merge-sortof the records contained in the sub-bucket files that match the filename pattern “PIL.j.*”. This pattern specifies all the sub-bucket filesthat comprise bucket j. Using appropriate POSIX system calls, commandsand APIs, this can be programmed so that the output of the merge-sortprocess is “piped” into, and read by, the inodes scanning thread. Whenthe Number_of_buckets is selected as noted above, all the DTRs can beread and held within a node fast memory (e.g., dynamic random accessmemory) and sorted without incurring additional file input/outputoperations.

The sub-bucket files for bucket j with names matching “PIL.j.*” are(typically) written by the different nodes during the parallel directoryscan and are gathered and read by one thread on one particular node.This can occur in parallel for several threads on many nodes, eachthread assigned a different bucket number, providing simultaneous,parallel and/or overlapping merge-sort executions and inodes scanningexecutions.

As such, DTRs are sorted into buckets, and the DTRs are sorted by inodenumber within each bucket. This allows reading read all the inodesneeded (in essentially sequential order and in parallel), with eachnode/thread taking on the job of reading a subsequence of inode numbers.For backup purposes, scanning a (sub)tree of the file system namespaceis performed using said parallel directory scan.

Fault Tolerance

Parallel inodes scanning allows fault tolerance in case of nodefailures. Should a node fail to perform an inodes scan of an assignedbucket j, the master node may re-assign the inodes scan of bucket j toanother processing node. Since there are typically many buckets, thecost of re-assigning a bucket is relatively small compared to the costof the entire job of processing all of the buckets. The paralleldirectory traversal also provides fault tolerance wherein each nodeperiodically writes checkpoint records, recording its progress inwriting output sub-bucket files. The master node retains copies of thework elements that are sent to each node, until said node checkpointsthe receipt of the work elements and sends an acknowledgement message toits master.

As will be appreciated by one skilled in the art, the present inventionmay be embodied as a system, method or computer program product.Accordingly, the present invention may take the form of an entirelyhardware embodiment, an entirely software embodiment (includingfirmware, resident software, micro-code, etc.) or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module” or “system.” Furthermore,the present invention may take the form of a computer program productembodied in any tangible medium of expression having computer-usableprogram code embodied in the medium.

Any combination of one or more computer usable or computer readablemedium(s) may be utilized. The computer-usable or computer-readablemedium may be, for example but not limited to, an electronic, magnetic,optical, electromagnetic, infrared, or semiconductor system, apparatus,device, or propagation medium. More specific examples (a non-exhaustivelist) of the computer-readable medium would include the following: anelectrical connection having one or more wires, a portable computerdiskette, a hard disk, a random access memory (RAM), a read-only memory(ROM), an erasable programmable read-only memory (EPROM or Flashmemory), an optical fiber, a portable compact disc read-only memory(CD-ROM), an optical storage device, a transmission media such as thosesupporting the Internet or an intranet, or a magnetic storage device.Note that the computer-usable or computer-readable medium could even bepaper or another suitable medium upon which the program is printed, asthe program can be electronically captured, via, for instance, opticalscanning of the paper or other medium, then compiled, interpreted, orotherwise processed in a suitable manner, if necessary, and then storedin a computer memory. In the context of this document, a computer-usableor computer-readable medium may be any medium that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.The computer-usable medium may include a propagated data signal with thecomputer-usable program code embodied therewith, either in baseband oras part of a carrier wave. The computer usable program code may betransmitted using any appropriate medium, including but not limited towireless, wireline, optical fiber cable, RF, etc.

Computer program code for carrying out operations of the presentinvention may be written in any combination of one or more programminglanguages, including an object oriented programming language such asJava, Smalltalk, C++ or the like and conventional procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The program code may execute entirely on the user's computer,partly on the user's computer, as a stand-alone software package, partlyon the user's computer and partly on a remote computer or entirely onthe remote computer or server. In the latter scenario, the remotecomputer may be connected to the user's computer through any type ofnetwork, including a local area network (LAN) or a wide area network(WAN), or the connection may be made to an external computer (forexample, through the Internet using an Internet Service Provider).

The present invention is described below with reference to flowchartillustrations and/or block diagrams of methods, apparatus (systems) andcomputer program products according to embodiments of the invention. Itwill be understood that each block of the flowchart illustrations and/orblock diagrams, and combinations of blocks in the flowchartillustrations and/or block diagrams, can be implemented by computerprogram instructions. These computer program instructions may beprovided to a processor of a general purpose computer, special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions, which execute via the processor ofthe computer or other programmable data processing apparatus, createmeans for implementing the functions/acts specified in the flowchartand/or block diagram block or blocks.

These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide processes for implementing the functions/actsspecified in the flowchart and/or block diagram block or blocks.

FIG. 6 shows a block diagram of example architecture of an embodiment ofa system 600 for implementing an embodiment of the invention. The system600 includes one or more processing or computing nodes 630 (such asnodes 12 in FIG. 1A). A processing node 630 includes a bus 602 or othercommunication mechanisms for communicating information, and a processor(CPU) 604 coupled with the bus 602 for processing information andprogram code. The node 630 also includes a main memory 606, such as arandom access memory (RAM) or other dynamic storage device, coupled tothe bus 602 for storing information and instructions to be executed bythe processor 604. The main memory 606 also may be used for storingtemporary variables or other intermediate information during executionor instructions to be executed by the processor 604. The processing node630 further includes a read only memory (ROM) 608 or other staticstorage device coupled to the bus 602 for storing static information andinstructions for the processor 604. A storage device 610, such as amagnetic disk or optical disk, is provided and coupled to the bus 602for storing information and instructions. The bus 602 may contain, forexample, thirty-two address lines for addressing video memory or mainmemory 606. The bus 602 can also include, for example, a 32-bit data busfor transferring data between and among the components, such as the CPU604, the main memory 606, video memory and the storage 610.Alternatively, multiplex data/address lines may be used instead ofseparate data and address lines.

The node 630 may be coupled via the bus 602 to a display 612 fordisplaying information to a computer user. An input device 614,including alphanumeric and other keys, is coupled to the bus 602 forcommunicating information and command selections to the processor 604.Another type of user input device comprises cursor control 616, such asa mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to the processor 604 andfor controlling cursor movement on the display 612.

The processing node 630 may be connected to other processing nodes viaan interconnect network 622 (such as interconnect 16 in FIG. 1A). Thesystem 600 further comprises multiple information storage nodes 624(such disk systems 14 in FIG. 1A), connected to the interconnect 622,wherein the processing nodes 630 and the information storage nodes 624can communicate therebetween.

According to one embodiment of the invention, the functions of theinvention are performed by a node 630 in response to the processor 604executing one or more sequences of one or more instructions contained inthe main memory 606. Such instructions may be read into the main memory606 from another computer-readable medium, such as the storage device610. Execution of the sequences of instructions contained in the mainmemory 606 causes the processor 604 to perform the process stepsdescribed herein. One or more processors in a multi-processingarrangement may also be employed to execute the sequences ofinstructions contained in the main memory 606. In alternativeembodiments, hard-wired circuitry may be used in place of or incombination with software instructions to implement the invention. Thus,embodiments of the invention are not limited to any specific combinationof hardware circuitry and software.

Various forms of computer readable media may be involved in carrying oneor more sequences of one or more instructions to the processor 604 forexecution. For example, the instructions may initially be carried on amagnetic disk of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to the node 630 can receivethe data on the telephone line and use an infrared transmitter toconvert the data to an infrared signal. An infrared detector coupled tothe bus 602 can receive the data carried in the infrared signal andplace the data on the bus 602. The bus 602 carries the data to the mainmemory 606, from which the processor 604 retrieves and executes theinstructions. The instructions received from the main memory 606 mayoptionally be stored on the storage device 610 either before or afterexecution by the processor 604.

The node 630 may also includes a communication interface 618 coupled tothe bus 602. The communication interface 618 provides a two-way datacommunication coupling to a network link 620 that is connected to theinterconnect 622. For example, the communication interface 618 may be anintegrated services digital network (ISDN) card or a modem to provide adata communication connection to a corresponding type of telephone line,which can comprise part of the network link 620. As another example, thecommunication interface 618 may be a local area network (LAN) card toprovide a data communication connection to a compatible LAN. Wirelesslinks may also be implemented. In any such implementation, thecommunication interface 618 sends and receives electricalelectromagnetic or optical signals that carry digital data streamsrepresenting various types of information.

The example versions of the invention described herein may beimplemented as logical operations in a distributed processing systemsuch as the system 600 including the nodes 630. The logical operationsof the present invention can be implemented as a sequence of stepsexecuting in the nodes 630, and, as interconnected machine modules,within the system 600. The implementation is a matter of choice and candepend on performance of the system 600 implementing the invention. Assuch, the logical operations constituting said example versions of theinvention are referred to for e.g. as operations, steps or modules.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other modifications mayoccur to those ordinarily skilled in the art.

What is claimed is:
 1. A method for managing a shared file system,comprising: performing traversal by multiple processing nodes inparallel of all directory paths and files in shared directories in theshared file system; partitioning identified directory traversal recordsinto ranges for the records in each range to be written independently inparallel by the multiple processing nodes; scanning file inodes inparallel, each inode comprising file attributes of files stored in saidshared directories and is assigned an inode number, and each range isassigned a range number; mapping each inode number to a range numberusing a mapping function; performing parallel execution threads in oneor more processing node utilizing local mutual exclusions to resolvecontention for access to local process resources, wherein during thescan of the file inodes, each processing node accesses a row of Nsub-ranges that logically comprises a complete range, wherein N is apositive number; and re-balancing a processing workload across themultiple processing nodes based on a monitored workload queue length ateach processing node by a master node.
 2. The method of claim 1, whereinthe complete range is defined by the mapping function, and sorts all thedirectory traversal records in those sub-ranges into a sequence orderedby inode number.
 3. The method of claim 1, further comprising:maintaining a local workload queue of elements on each processing node,the elements representing a plurality of un-read shared directories;reporting workload queue length by each processing node to the masternode; and monitoring the processing workload and workload queue lengthof the multiple processing nodes performing the directory traversal bythe master node, wherein re-balancing the processing workload across themultiple processing nodes comprises determining whether each processingnode is under-utilized or over-utilized by comparing the monitoredworkload queue length with a predetermined queue length threshold, andredistributing work elements from over-utilized processing nodes tounder-utilized processing nodes.
 4. The method of claim 1, wherein upona node failing to perform an inode scan of an assigned range, the masternode reassigns the inode scan of the assigned range to anotherprocessing node.
 5. The method of claim 4, wherein each nodeperiodically writes checkpoint records for recording progress in writingoutput of sub-range files, and the parallel execution threads use localmutual exclusions for resolving contention for access to a plurality oflocal process resources.
 6. The method of claim 5, wherein the masternode retains copies of work elements sent to each node until a nodecheckpoints receipt of the work elements and sends an acknowledgementmessage to the master node.
 7. The method of claim 1, wherein saidscanning comprising reading full storage units of file inodes from eachof the plurality of ranges of directory traversal records, wherein thefile inodes represent all the files in the shared file system in aninode number order and are stored in an inode file, such that the fileinodes read are specified by said records, wherein each set of recordscomprising a range is sorted by inode number.
 8. The method of claim 1,wherein the function comprises a hash function that inputs integers andmaps the integers to ranges.
 9. A computer program product for managinga shared file system, the computer program product comprising a computerreadable storage medium having computer usable program code embodiedtherewith, the program code readable/executable by one or moreprocessing nodes to: perform directory traversal, by multiple processingnodes, in parallel of all directory paths and files in shareddirectories in the shared file system; partition, by the multipleprocessing nodes, identified directory traversal records into ranges forthe records in each range to be written independently in parallel by themultiple processing nodes; scan, by the multiple processing nodes, fileinodes in parallel, each inode comprising file attributes of filesstored in said shared directories and is assigned an inode number, andeach range is assigned a range number; map, by the multiple processinginodes, each inode number to a range number using a mapping function;perform, by the multiple processing nodes, parallel execution threadsutilizing local mutual exclusions to resolve contention for access tolocal process resources, wherein during the scan of the file inodes,each processing node accesses a row of N sub-ranges that logicallycomprises a complete range, wherein N is a positive number; andre-balance a processing workload across the multiple processing nodesbased on a monitored workload queue length at each processing node by amaster processing node.
 10. The computer program product of claim 9,wherein the complete range is defined by the mapping function, and sortsall the directory traversal records in those sub-ranges into a sequenceordered by inode number.
 11. The computer program product of claim 10,further comprising program code to: maintain, by the multiple processingnodes, a local workload queue of elements at each processing node, theelements representing a plurality of un-read shared directories; reportworkload queue length, by the multiple processing nodes, to the masterprocessing node; monitor, by the multiple processing nodes, processingworkload and workload queue length of the multiple processing nodesperforming the directory traversal by the master processing node,wherein re-balance the processing workload across the multipleprocessing nodes comprises determining whether each processing node isunder-utilized or over-utilized by comparing the monitored workloadqueue length with a predetermined queue length threshold, andredistributing work elements from over-utilized processing nodes tounder-utilized processing nodes.
 12. The computer program product ofclaim 9, wherein upon a node failing to perform an inode scan of anassigned range, the master node reassigns the inode scan of the assignedrange to another processing node.
 13. The computer program product ofclaim 12, wherein each node periodically writes checkpoint records forrecording progress in writing output of sub-range files.
 14. Thecomputer program product of claim 13, wherein the master node retainscopies of work elements sent to each node until a node checkpointsreceipt of the work elements and sends an acknowledgement message to themaster node.
 15. A system for managing a shared file system, comprising:a plurality of storage nodes; and a plurality of processing nodescoupled to the information storage nodes, each processing node performstraversal in parallel of all directory paths and files in shareddirectories of the file system, partitions identified directorytraversal records into ranges for the records in each range to bewritten independently in parallel by the plurality of processing nodes,scan file inodes in parallel by the plurality of processing nodes, eachinode comprising file attributes stored in said shared directories, eachinode is assigned an inode number, and each range is assigned a rangenumber, and map each inode number to a range number using a mappingfunction, wherein during the scan of the file inodes, each processingnode accesses a row of N sub-ranges that logically comprises a completerange, and the plurality of processing nodes maintain a local workloadqueue of elements at each processing node, perform parallel executionthreads utilizing local mutual exclusions to resolve contention foraccess to local process resources, wherein traversal processing workloadand reporting workload queue length of the plurality of processing nodesis monitored by a master processing node that re-balances the processingworkload across the plurality of processing nodes.
 16. The system ofclaim 15, wherein the complete range is defined by the mapping function,and sorts all the directory traversal records in those sub-ranges into asequence ordered by inode number, wherein N is a positive number. 17.The system of claim 16, wherein the elements representing un-read shareddirectories, and reporting workload queue length by each processing nodeto the master processing node, wherein the master processing nodere-balances the processing workload across the plurality of processingnodes based on comparing the monitored workload queue length with apredetermined queue length threshold, wherein re-balancing of theprocessing workload by the master processing node comprises determiningwhether each processing node is under-utilized or over-utilized, andredistributing work elements from over-utilized processing nodes tounder-utilized processing nodes.
 18. The system of claim 17, whereinupon a processing node failing to perform an inode scan of an assignedrange, the master node reassigns the inode scan of the assigned range toanother processing node.
 19. The system of claim 18, wherein eachprocessing node periodically writes checkpoint records for recordingprogress in writing output of sub-range files, and the master noderetains copies of work elements sent to each processing node until aprocessing node checkpoints receipt of the work elements and sends anacknowledgement message to the master node.